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DESCRIPTION 

SIGHT-LINE DETECTION METHOD AND DEVICE, AND THREE- 
DIMENSIONAL VIEW-POINT MEASUREMENT DEVICE 
Technical Field 

5 [0001] A line-of-sight detection method and device according to the 
present invention relate to a line-of-sight detection method and line-of- 
sight detection device that can minimize postural limitations of a subject 
and detect a line-of-sight without attaching markers, and a three- 
dimensional view-point measurement device relates to a device that 

10 measures a three-dimensional view-point by simultaneously detecting 
line-of-sight directions of human both eyes. 
Background Art 

[0002] Since line-of-sight detection methods and line-of-sight detection 
devices have been expected to be used for a wide range of application 

15 such as medical inspection and man-machine interfaces (line-of-sight 
input), many proposals have been made. Most of them use a basic 
principle of detecting a line-of-sight by using a corneal reflection point 
and a pupil center. The invention described in Patent Document 2 is 
one relating to the aforementioned detection and is directed to detecting 

20 feature points. The invention described in Patent Document 3 makes 
an in-depth proposal for measurement that takes a shape of cornea into 
account. 

[0003] In addition, many line-of-sight detection devices that have been 
proposed can roughly be divided into a type that mainly fixes a head of 
25 a subject and another type that mounts a detection device on the head of 
the subject. Both types of devices have hindered everyday behavior of 
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the subject seriously and put an enormous load on the subject when 
performing line-of-sight measurement. Further, recently, non-contact 
type has been designed for the purpose of reducing the load on the 
subject, but such type of detection devices is only placed remotely from 

5 the subject and has a very limited measuring range, virtually 
immobilizing the subject for measurement. A method for detecting a 
line-of-sight from an image of eyes using image processing etc., which 
has been proposed as a method for eliminating restrictions on movement 
of the subject, has poor time characteristics requiring a long time for 

10 image processing. 

Conventional detection devices also have shortcomings of low precision 

» 

because the same camera used for face detection is also used for line-of- 

■ 

sight detection. To solve this problem, a non-contact line-of-sight 
measurement device described in Patent Document 1 requires a 

15 spectacle frame with three markers attached on a face. If the 
spectacles are attached, general glasses for vision correction cannot be 
put on. Also for a user or subject who usually does not wear glasses, 
the spectacles for measurement become a burden. 
[0004] Background technology for the three-dimensional view-point 

20 measurement device will be described below. 

-In Patent Document 4 (Japanese Patent No. 2739331: ATR), a 
technology is described that detects a pupil by using retinal reflection, 
captures a three-dimensional position of a pupil center 30 from face 
images taken by two cameras, and determines the position by 

25 triangulation. This can be considered to be detecting a three- 
dimensional position of a pupil. Moreover, illumination light of 
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different wavelengths must be prepared. 

-In the Patent Document 1 (Japanese Patent Application Laid-open No. 
Hei 10-066678: NTT, NAC), a technology is described that attaches 
three markers on a face, detects an eyeball position using a stereo 
5 camera having two wide cameras, and tracks the eyeball using one 
narrow camera. 

-In Patent Document 5 (Japanese Patent No. 2988235: Nissan Motor 
Co., Ltd.), a technology is described that determines a center of corneal 
sphere from corneal reflection images with a plurality of illuminations 
10 and determines a line-of-sight direction from the center of corneal 
sphere and a pupil center. 

-In Patent Document 6 (Japanese Patent No. 3324295: Nissan Motor 
Co., Ltd.), like the Patent Document 5, a technology is described that 
determines a center of corneal sphere from corneal reflection images 
15 with a plurality of illuminations and determines a line-of-sight direction 
from the center of corneal sphere and a pupil center. These 
technologies are all intended strictly for detecting the line-of-sight 
direction and do not determine the position of a view-point to which the 

* 

line-of-sight is directed. 
20 Patent Document 1 : Japanese Patent Application Laid-open No. Hei 10- 
66678 

Patent Document 2: Japanese Patent Application Laid-open No. Hei 11- 

■ 

56782 

Patent Document 3: Japanese Patent Application Laid-open No. 2002- 
25 102172 

Patent Document 4: Japanese Patent No. 273933 1 
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Patent Document 5: Japanese Patent No. 2988235 
Patent Document 6: Japanese Patent No. 3324295 
Disclosure of the Invention 

[0005] A first object of the present invention is to provide a line-of- 
5 sight detection method that can detect a line-of-sight while minimizing 
postural limitations of a subject and without attaching markers, and a 
line-of-sight detection device for realizing the line-of-sight detection 
method. 

A second object of the present invention is to provide a three- 

* 

10 dimensional view-point measurement device for measuring a three- 
dimensional view-point by simultaneously detecting line-of-sight 

» # 

directions of both eyes of a person. 

[0006] To achieve the first object, a line-of-sight detection method of a 
subject described in claim 1 according to the present invention using: 

15 a first camera for measuring the position of a pupil relative to a 

coordinate system; a second camera having a light source arranged at a 
known position in the coordinate system and forming a corneal 
reflection point to obtain data of a size of vector r from the corneal 
reflection point to a pupil center and an angle <j> of the vector r relative 

20 to a coordinate axis of the coordinate system; and a calculation means 
for calculating the line-of-sight direction for executing steps below 
based on information from each of the cameras, comprises the stages of: 
determining a relational formula, including the steps of: 
obtaining data on a coordinate point O of the position of a pupil 

25 of a subject with the first camera by making the subject gaze at a known 
point G in the coordinate system; 
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obtaining, in the state of the subject, data of the corneal 
reflection point, a size of vector r from the reflection point to a pupil 
center P, and an inclination of the vector r relative to the coordinate 
axis with the second camera; 

■ 

5 calculating an angle 0 between a line connecting a reference 

position of the second camera and the pupil center and a line-of-sight of 
the subject by the calculation means; and 

calculating a formula 0 = f (r*) showing a relationship between 
r* related to r and 0 based on the measured values and calculated value; 
10 and 

determining a line-of-sight, including the steps of: 
obtaining data on a coordinate point 0' of the pupil position of 
the subject with the first camera by making the subject gaze at an 
unknown point G f in the coordinate system; 
1 5 obtaining data of the corneal reflection point, a size of vector r' 

from the reflection point to the pupil center P, and an inclination <j>' of 
the vector r 1 relative to the coordinate axis with the second camera; and 

calculating 0' = f (r*') by using the relational formula to obtain 
the unknown point G f from the inclination <j>' and 0 f . 
20 [0007] In the method described in claim 2 according to the present 
invention, r* in the method described in claim 1 is r itself or a corrected 
value of r based on OP, and r*' is r 1 itself or a corrected value of r 1 based 
on OF. 

In the method described in claim 3 according to the present invention, 
25 the first camera is a stereo camera arranged by aligning a baseline in a 
horizontal axis direction of the coordinate system, and a light source of 
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the second camera is constructed so as to provide an optical axis that is 
substantially aligned with that of the second camera. 
In the method described in claim 4 according to the present invention, 
the formula 8 = f (r*) showing the relationship between r* and 0 is 
given by 0 = k x r* (where k is a constant). 

In the method described in claim 5 according to the present invention, 
the pupil is one of the pupils of the subject. 

[0008] A device described in claim 6 according to the present invention 
comprises: 

a first camera for measuring a position P of a pupil relative to the 
coordinate system; 

a second camera having a light source arranged at a known 
position in the coordinate system to obtain data of a size of vector r 
from a corneal reflection point to a pupil center illuminated by the light 
source and an angle $ of r relative to the coordinate axis; and 

a calculation means for executing the steps of: 

obtaining data on a coordinate point P of the position of a pupil 
of a subject with the first camera by making the subject gaze at a known 
point G in the coordinate system; 

obtaining, in the state of the subject, data of the corneal 
reflection point, a size of vector r from the reflection point to a pupil 
center P, and an inclination <|> of the vector r relative to the coordinate 
axis with the second camera; 

calculating an angle 0 between a line connecting a reference 
position of the second camera and the pupil center and the line-of-sight 
of the subject and calculating a formula 0 = f (r*) showing a relationship 
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between r* related to r and 0; 

obtaining data on a coordinate point O 1 of the pupil position of 
the subject with the first camera by making the subject gaze at an 
unknown point G 1 in the coordinate system; 

obtaining data of the corneal reflection point, a size of vector r' 

■ 

from the reflection point to the pupil center P, and an inclination <J> ! of 
the vector f relative to the coordinate axis with the second camera; and 

calculating 9 1 = f (r* 1 ) from r* f related to r' by using the relational 
formula to further obtain the unknown point G' from <J)' and 0\ 
[0009] To achieve the second object, a three-dimensional view-point 
measurement device described in claim 7 according to the present 
invention comprises: 

two cameras, a first light source arranged near one of the two 
cameras, a second light source arranged near another of the two cameras, 
a control means for controlling ON/OFF of the first light source and the 
second light source and obtaining an image signal in sync with ON/OFF, 
and a calculation means for extracting a pupil and corneal reflection 
from the obtained image signal 

A device described in claim 8 according to the present invention 
comprises: 

two cameras, a first light source arranged near one of the two 
cameras, a second light source arranged near another of the two cameras, 
a control means for controlling ON/OFF of the first light source and the 
second light source and obtaining an image signal in sync with ON/OFF, 
and a calculation means for extracting a pupil and corneal reflection 
from the obtained image signal and calculating a line-of-sight vector 
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from these positions. 

§ 

A device described in claim 9 according to the present invention 
comprises: 

two cameras, a first light source arranged near one of the two 
cameras, a second light source arranged near another of the two cameras, 
a control means for controlling ON/OFF of the first light source and the 
second light source and obtaining an image signal in sync with ON/OFF, 
and a calculation means for extracting a pupil and corneal reflection 
from the obtained image signal and calculating a three-dimensional 
position of the pupil from these positions. 

The device described in claim 10 according to the present 
invention is any of the devices described in claims 7 to 9, wherein: 

the first light source and die second light source are configured 
to have an approximately identical emission wavelength. 
[0010] As has been described in detail, a line-of-sight detection method 
of the present invention is a line-of-sight detection method of a subject 
that uses a first camera for measuring the position of a pupil relative to 
the coordinate system, a second camera having a light source arranged 
at a known position in the coordinate system and forming a corneal 
reflection point to obtain data of a size of vector r from the corneal 
reflection point to the pupil center and an angle § of the vector r relative 
to the coordinate axis, and a calculation means for calculating a line-of- 
sight direction and for executing the steps below based on information 
from each of the cameras. 
In the stage of determining a relational formula, 
a formula 0 = f (r*) showing a relationship between r* related to r 
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(vector from the corneal reflection point to the pupil center) and 0 
(angle between the line connecting the second camera and the pupil 
center and the line-of-sight of the subject) is calculated in advance based 
on measured values by using the first camera and second camera and a 
calculated value by the calculation means. Then, in the stage of 
determining a line-of-sight, & = f (r* f ) is calculated using the 
aforementioned relational formula and an unknown point G' is obtained 
from an inclination (j)' measured in this stage and 9\ 
Therefore, the line-of-sight can be measured without severely 
constraining a subject. 

Since the stage of determining the relational formula is performed for a 
specific subject and measurement is performed using a relational 
formula obtained in the stage, there is no room for occurrence of 
measurement error due to an individual difference of the subject. 
Consequently, correct data on the line-of-sight can be obtained. 
[0011] The three-dimensional view-point measurement device of the 
present invention can measure a three-dimensional view-point by 
simultaneously detecting the line-of-sight directions of both eyes of a 
person. Like the aforementioned method, there is no need for severely 
constraining a person. 
Brief Description of the Drawings 

[0012] Fig. 1 is a schematic diagram showing an arrangement of 
embodiments of apparatuses for carrying out a line-of-sight detection 
method according to the present invention. An enlarged view of an 
example of a physical relationship between a pupil and a corneal 
reflection (point) is shown. 
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Fig. 2 is a block diagram showing a configuration of apparatuses and 
the like for carrying out the line-of-sight detection method of the present 
invention. 

Fig. 3 is a flow chart for describing the line-of-sight detection method 
according to the present invention. 

Fig. 4 is a perspective view showing the arrangement of apparatuses in 
relation to a world coordinate system. 

Fig. 5 is a graph illustrating mapping between a pupil and a virtual 
view-point plane [correspondence between (r, <|>) and (| 0 | =k | r |, -<|>)]. 
Fig. 6 is a perspective view for illustrating the relationship between a 
plane to be viewed and a virtual view-point plane (H). 
Fig. 7 is a diagram for illustrating meanings of 6 and $ in the virtual 
view-point plane (H). 

Fig. 8 is a graph showing the relationship between a direction vector 
and an angle. 

Fig. 9 is a diagram showing the arrangement of two cameras and a 
subject. 

Fig. 10 is a diagram showing a correspondence between turning 
ON/turning OFF of a light source and a bright/dark pupil. 
Fig. 11 is a diagram showing turning ON of a light source 105 and 
images of pupils of a subject. 

Fig. 12 is a diagram showing turning ON of a light source 106 and 
images of pupils of a subject. 

Fig. 13 is a diagram showing a whole face and a range to be tracked. 
Fig. 14 is a schematic diagram for determining a view-point by line-of- 
sight detection of both eyes. 
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Fig. 15 is a diagram showing an association between a line-of-sight 
vector, and <j>, 0. 

Fig. 16 is a diagram for determining a line-of-sight vector from a pupil 
center and a corneal reflection. 
5 Fig. 17 is a schematic diagram for detecting a three-dimensional 
position of a pupil and controlling the size and position of an image to 
be shown on a screen. 

Fig. 1 8 is a diagram showing how a field angle of an image changes as 
the position of a pupil changes. 

10 Figs. 19(a) and 19(b) are diagrams showing the reflection law. 

Figs. 20(a) and 20(b) are diagrams showing the cases in which a 
spherical surface moves in a lateral direction relative to a camera and a 
light source. Fig. 20(a) is the case in which the position of the camera 
and that of the light source are different. Fig. 20(b) is the case in 

15 which the position of the camera and that of the light source are same. 

Figs. 21(a) and 21(b) are diagrams showing the spherical surface 
moving in a depth direction. Fig. 21(a) is the case in which the 
position of the camera and that of the light source are different. Fig. 
21(b) is the case in which the position of the camera and that of the light 

20 source are same. 

Fig. 22 is a diagram showing a pupil moving by maintaining a constant 
distance from the camera and light source. 

Figs. 23(a) and 23(b) are diagrams showing the relationship between a 
change of the line-of-sight vector and the corneal reflection when the 
25 position of the pupil center is fixed. 

Fig. 24 is a diagram showing the relationship between the angle 0 
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formed by a camera-pupil vector and the line-of-sight vector and a size 
|r| of a corneal reflection center-pupil center vector r in a camera image. 
Fig. 25 is a diagram showing an effect of a change in the distance 
between the camera and the pupil center on |r|. 

Fig. 26 is a diagram showing an example in which a correction is made 

so as to change the size of an eyeball in accordance with the change in 

the distance between the camera and the pupil center. 

Explanations of Numerals 

[0013] 10, 1 1 : First camera 

12: Second camera 

14: Camera interface 

15: Lighting control unit 

16: Camera driving device 

20: Bus line 

22 : Processing unit (CPU) 

* 

24: Input device 
26: Storage device 
30: Output interface 
3 1 : Output device 
101, 102: camera 
103, 104: Lens 
105. 106: Light source 
107: Face 
108: Pupil 

109: Corneal reflection 
110: Camera 
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111: Screen 
120: Camera 
121: Light source 

■ 

122: Contact point between a tangent plane of a spherical surface and 
5 the spherical surface 

123: Center of a spherical surface 
124: Spherical surface 
125: Tangent plane of a spherical surface 
126: Eyeball 

1 0 Best Modes for Carrying Out the Invention 

[0014] Best modes for carrying out a line-of-sight detection method and 
device of the present invention will be described in detail below with 
reference to the drawings. 

Fig. 1 is a schematic diagram showing an arrangement of an 
15 embodiment of apparatuses for carrying out a line-of-sight detection 
method of the present invention. Fig. 4 is a perspective view showing 
the arrangement of apparatuses for carrying out die line-of-sight 
detection method of the present invention in relation to the world 
coordinate system. 

20 A camera for eyeball shooting with high magnification is used as a 
second camera 12. This camera is arranged in this embodiment at 
origin of the world coordinate system, at O (0, 0) in Fig. 4. In Fig. 4, 
an illustration of the camera 12 is omitted. 

Fig. 1 shows an example of the physical relationship of the corneal 
25 reflection (point) based on data obtained with the second camera 12 by 

» 

enlarging it. This image is displayed by an output device described 
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later. 

First camera 10, 11 is a stereo camera composed of two cameras and for 
detecting a three-dimensional position of a pupil. 
The distance (base length) between the first cameras 10, 11 is parallel to 
the X axis of the world coordinate system. 

The second camera 12 is sighted at least on one pupil and shown in Fig. 
1 by enlarging it. Data of the corneal reflection point and the pupil 
center is obtained. 

An illumination means (not shown) is provided integrally or in 
connection with the first camera 10, 11 and the second camera 12 
respectively. 

[0015] Fig. 2 is a block diagram showing the configuration of 
apparatuses for carrying out the line-of-sight detection method of the 
present invention. 

Image output obtained by CCDs of the first camera 10, 11 and the 
second camera 12 is connected to a bus line 20 via an interface 14. 
In this embodiment, the second camera 1 2 has a light source provided at 
a known position O in the coordinate system and forming a corneal 
reflection point to obtain data of the size of vector r from the corneal 
reflection point to the pupil center and the angle § of the vector r 
relative to the coordinate axis. In this embodiment, an optical axis of 
light from the light source is provided so that the optical axis is aligned 
with that of the second camera 12. 

The second camera 12 is supplied by a signal supplied from a 
processing unit (CPU) 21 via a driving device 16 and is sighted 
(directed and focused) on a target eye. 
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* 

An operator operates an input device 24 for sighting when necessary 
while viewing an eye image (See Fig. 1) displayed in a display screen 
forming part of an output device 31. The illumination means (not 
shown) for each camera is operated by a signal from a lighting control 
5 unit 15. 

In a storage device 26, programs for executing control described later 
and a RAM area for performing operations are provided. Image 

■ 

information of a subject, operation result information, system operation 
information and so on are output to the output device 3 1 via an output 
10 interface 30. The output device 31 includes an image display device, a 
printer and so on. 

[0016] The first camera 10, 11 is provided to detect a three-dimensional 
position on coordinates of the pupil center of eye. The second camera 
1 2 is a camera for detecting a line-of-sight with high magnification and 
15 captures only an image of a surrounding eye including a pupil. 

An automatic tracking control means (driving device 1 6) is provided to 
track one or both of two eyes based on three-dimensional pupil position 
information from the first camera. 

It is also possible to manually align the optical axis of a camera. As 
20 will be described later, a line-of-sight vector is determined from the 
three-dimensional pupil position P obtained from output of the first 
camera 10, 11, and the position of the center of a corneal reflection 
image and the pupil center P obtained from output of the second camera 
12. 

25 [0017] (About illumination light sources) Illumination light sources to 
be prepared can include ones arranged in or near an aperture of a 

15 
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camera lens (hereinafter referred to as inside light sources) and ones 
arranged apart from the aperture (hereinafter referred to as outside light 
sources). In images of inside light sources, pupils tend to appear 
brighter than other parts of a face. This is described in connection with 
5 Fig. 13B in Japanese Patent No. Hei 7-82539. 

Conversely, pupils tend to appear darker than other parts due to outside 
light sources. This is described in connection with Fig. 13C of the 
same. 

By synchronizing the inside and outside light sources with a video 
10 signal to turn on alternately and producing an image difference by 
taking an obtained latter image from an obtained former image in real 
time, other parts than the pupils disappear by canceling out each other, 
making it easier to detect only the pupils. Such a light source is 
provided for each camera. 
15 [0018] In this embodiment, light sources are mounted for total three 
cameras of the first and second cameras. 

In order to prevent light emitted by light sources of other cameras and 
reflected by the spectacles from being taken in an image as a spectacle 
reflection image due to this configuration, light sources of different 
20 center wavelengths are used as light sources to be mounted on each 
camera and a band-pass filter with each wavelength as the center 
wavelength is mounted on each camera. 

In this case, when generally an emission wavelength band of light 
source has a wide band, the emission wavelength overlaps in three or 
25 more light sources, and therefore the light sources of each other's 
camera interfere even if the selected band-pass filter has a narrow band. 
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To prevent this, a band-pass filter is placed before a light source, the 
wavelength band of light caused to be emitted to a face is narrowed, 
overlapping of the emission wavelength band is reduced as much as 
possible, and further before each camera a band-pass filter is placed 
5 which has almost the same center wavelength as that of the band-pass 
filter placed before the light source of the same camera so that mutual 
interference can be eliminated. 

[0019] A line-of-sight detection method of the present invention will be 
described below with reference to Figs. 3 and 4. The line-of-sight 
10 detection method according to the present invention comprises a stage 
of determining a relational formula for determining a relational formula 

* 

and a stage of measuring a line-of-sight for determining any line-of- 
sight of a subject based on the determined relational formula. 
The stage of determining a relational formula includes the steps 0 to 10 
15 and the stage of measuring a line-of-sight includes the steps 1 to 10. 

[0020] Fig. 4 shows an arrangement of apparatuses and the like and 
related planes and the like for carrying out the line-of-sight detection 
method of the present invention in connection with the world coordinate 
system. 

20 [Stage of determining a relational formula] 

(Step 0) Assignment of the position of each camera relative to the world 
coordinate system. 

Define the origin of the world coordinate system as O (0, 0) and arrange 

* 

the first camera. 
25 (Step 1 ) Make a subject gaze at a known point. 

Make a subject gaze at a point Q (xq, yq, Zq) whose coordinates in the 
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world coordinate system are known. 

(Step 2) Acquisition of coordinate position data of the pupil center P of 
the subject with the second camera. 

Determine a three-dimensional coordinate P (x P , y P , z P ) of the pupil 
5 center with a stereo camera. 
(Step 3) Calculation of |OP| 
Determine a direction vector 

(l x , l y , l z ) = (x P - xo, yp - yo, z P - zo) of a vector OP and its length 

|OP| = [(x P - xo) 2 + (y P - yo) 2 + (z P - 2cf] m 
10 (Step 4) Calculation of a virtual view-point plane H that is perpendicular 

to |OP| and passes through O 
The plane H is given by the following formula: 
(l x -x + l y -y + l z -z)-(lx-xo + ly-yo+lz-zo) = 0 (1) 
(Step 5) Calculation of a vector PQ representing a line-of-sight 
1 5 A direction vector (s x , s y , S*) of PQ representing the line-of-sight is 
given by (s x , s y , s^ = (x Q - x P> y Q - y P , zq - z P ) and a straight line PQ is 
given, because the line passes through P (x P , yp, zp), by the following 
formula: 

■ 

(x - x P )/s x = (y - y P )/sy = (z - z P )/Sz (2) 
20 (Step 6) Calculation of a coordinate G (xq, yo, z G ) of an intersection of 
the straight line PQ and the virtual view-point plane H 
(x, y, z) determined by setting the aforementioned formula (2) = t is 
given by the following formula (3): 
x = s x t + Xp 

25 y = Syt + y P (3) 

Z = S z t + Zp 
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Substituting (3) into the formula of the plane H determines t: 

t = (l x 2 + 1/ + l 2 2 )/(lxS x + lySy + (4) 

Substituting (4) into (3) yields the intersection G (x G > yo, zg) in the 
world coordinate system. 
5 (Step 7) Calculation of an angle 9 between the vector OP and vector PG 
Determine an angle 6 formed by the direction vector (l x , l y? 1 2 ) of the 
vector OP and the direction vector PG (s G , s G > s G ) of the line-of-sight 
vector. 

9 = cos 1 [|U + l y s y + \& |/(l x 2 + l y 2 + ] z 2 ) m • (s x 2 + s y 2 + s z 2 ) m ] > 0 (5) 
10 (Step 8) Measurement of r and ty a 

Obtain the coordinates of corneal reflection center and pupil center with 
the first camera (first camera 20) and determine the size of vector r from 
the corneal reflection center to the pupil center and angle 0 (relative to 
the coordinate axis). 

15 Assuming that the coordinates of corneal reflection center and pupil 
center detected from an image of the camera for line-of-sight detection 
(second camera 12) are (g x , g y ) and (p x , p y ) respectively, 
the distance and angle can be calculated as shown below: 
If Irk 0, 

20 let |r| = [(p x - g x ) 2 + (p y - gy ff n (6) 
and <j> a = cos' 1 ^ - g y )/|r| (7) 
If putting <t> a ' (0 < <j) a ' < n), 

Ifpx<gx,<l>a = K 
If Px > gx, <i>a = "K 

25 When a subject gazes at Q, determine |r| as |rQ| (in units of pixels). 
(Step 9) Determine |r*| by correcting |r| with OP. 
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The second camera 12 detects the center of a corneal reflection 
image and the pupil center image, and from their relative physical 
relationship estimates an angle 6 formed by the vector OP and a line-of- 
sight PG 

5 In this case, when images have different zoom ratios even while 
viewing in the same direction, a relative relation between the center of a 
corneal reflection image and the pupil center image changes, and thus 
the zoom ratio of image is an important element. 
To solve this problem, first a relation is measured between the distance 

10 and zoom ratio in advance to determine a relational formula. Since die 
three-dimensional position of a pupil is known, the distance is 
calculated between the camera for line-of-sight detection and a pupil, 
and then the zoom ratio is calculated using the relational formula to 
correct the distance between the center of a corneal reflection image and 

15 the pupil center image. 

[0021] (Step 10) Determine a relational formula f between |r*| and 9 by 
using 0 and |r*|. 

Let 9 and |r| when a subject gazes at O be 9 0 and |r 0 | respectively, 

since | r* 0 1 = 0 when |r 0 | = 0, assume a linear relation 
20 9 = k|r*| (8) 

between a value 9 when Q is viewed by the subject and |r*| and 

determine a coefficient k. 

Let k = 9Q/|r* Q | (9) 

and <j) a = -V (10) 
25 By determining such the coefficient k (generally a relational formula f 

between |r*| and 9), the line-of-sight is calibrated. 

20 
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|r*| is a function of 0 and is independent of (J> a . This is because it is 
assumed that k is constant in spite of the value of <|>a or <J> a ' . 
This will be further described with reference to Fig. 5. When mapping 
from (|r*|, (jO to (|9|, <|>a), the origin is mapped to the origin and mapping 
is simply scaled and right and left are reversed. 

[0022] The relational formula obtained in the aforementioned stage of 
determining a relational formula is used [stage of measuring a line-of- 
sight], 

(Step 1) Sighting on a subject by the first camera if required 

The view point of the subject is assumed to be an unknown Q f (x Q , y Q , 

zq) of the plane to be viewed. 

An object of a procedure below is to determine this coordinate. 

(Step 2) Measurement of three-dimensional coordinate of the pupil 

center 

As shown Fig. 5, measure coordinate position data (xp, yp, zp) of the 
pupil center P of a subject using the first camera 10, 12. 
(Step 3) Calculation of |OP| 
Determine a direction vector 

(l x > ly, lz) = (xp - *o> yp - yo, z P - zo) of a vector OP 1 and its length 
|OP| = [(x P - xo) 2 + (y P - y 0 ) 2 + (zp - zq) 2 ] 1/2 

(Step 4) Measure the size of vector f and angle relative to the 
coordinate axis by obtaining the coordinates of the center of corneal 
reflection and the pupil center with the first camera. 
If |r'| * 0, 

letlr'i^^x-gx^ + ^-gy) 2 ] 172 (6') 
and 4b = cos" l (p y - g y )/|r| (7) 
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If putting <J> a ' (0 < <j> b ' < 

■ 

IfPx<gx, ^b^b 
IfPx>gx, 4>b = *4>b 

(Step 5) Determine r*' by correcting r' with |OF|. 

(Step 6) Determine 9' using the relational formula determined in the 

previous stage. 

e' = f(|r*l) = k|r*'| 

(Step 7) Determine a line-of-sight vector G\ 

Take an orthogonal coordinate system (with the origin aligned with an 
absolute coordinate system) by selecting the virtual view-point plane H 
as an x' - y' plane, as shown in Fig. 6 (The z' axis passes through the 
point P). 

Assume that the x' and y* axes on the plane H are rotated by a in the 
horizontal direction and by p in the vertical direction relative to the x 
and y axes in the absolute coordinate system. 

Fig. 7 (left) shows a view of the plane H from the z' axis direction and 
Fig. 7 (right) shows a plane that passes three points of O, G', and P. 
The point G' in the orthogonal coordinate system of the virtual view- 
point plane H is determined by 6, §\>, and |OP|. 
As is evident from Fig. 7 (right), 
|OG'| = |OP|tan0 and 

xg\ yc of the point G' (x©, y&, 0) in the orthogonal coordinate system of 

the virtual view-point plane H is determined, if ^> = <|>b , by 

x G - = lOG'Icos^' 

yc = PG'IsinK 

(Step 8) Determine PG\ 
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Since the coordinate of the point P in the orthogonal coordinate system 
of the virtual view-point plane H is (|OP|, 0, 0), a line-of-sight vector in 
the same coordinate system is determined, as a vector PG* connecting 
from P to G', by 
PG = (x G ., y<y - |OP|) 

This is also a direction vector of the line-of-sight vector in the 

* r 

orthogonal coordinate system of the virtual view-point plane H. 
(Step 9) Calculation of a direction vector of the line-of-sight vector 
The direction vector (1^, l y , l z ) of the vector OP in the world coordinate 
system agrees with the z' coordinate of the orthogonal coordinate system 
of the virtual view-point plane H. If the x' axis in the orthogonal 
coordinate system of the virtual view-point plane H is rotated by -a 
around the origin so that the x' axis agrees with the x axis of the world 
coordinate system and the y' axis is rotated by -p around the origin so 
that the y* axis agrees with the y axis, the orthogonal coordinate system 

■ 

of the virtual view-point plane H and the world coordinate system agree. 
The direction vector of the vector PG 1 in the orthogonal coordinate 
system of the virtual view-point plane H after the aforementioned 
coordinate rotation agrees with the line-of-sight vector in the world 
coordinate system. 
Where Formula (11) 



After the rotation, the direction vector (Sx, s y , s 2 ) of the line-of-sight 



cosa = 





(id 
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10 



15 



20 



vector in the world coordinate system is given by the formula (12) 



shown below. 
Formula (12) 



\ s y, 



/cos/? 
0 

, sin/J 



0 
1 
0 



-sin/5 
0 

cos/5 



I cosa -sin a 0\ 
sina cosa 0 
0 0 1 



f-\OP\\ 



Xq, 



(12) 



Note that Fig. 8 shows relations between the orthogonal coordinate 
system and angles. 

(Step 1 0) Calculation of a view-point Q' 

Using (Sx, Sy, s z ) determined in the previous step, a line-of-sight formula 

in the world coordinate system is represented by 

(x - x P )/s x = (y - y P )/Sy - (z - z P )/s 2 (=t) (2) 

If the visual object is a plane and its formula is given by 

m x -x + m y -y + m x 'z+d = 0 (13) 

an intersection of the formulas (1 1) and (12) is Q and it is determined. 
From (2) 

X = Sxt + X P 

y = Syt + y P (3) 
z = Szt + z P 

Substituting (3) into the formula of the plane H yields 
Formula (14) 



t = 



d - {m x x p + m y F + m z z P ) 



m x s x + m y s y + m z s z 



(14) 



Substituting (14) into (3) determines the view-point Q' (xqj, y Q -, zq-). 
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[0023] Different modifications of embodiments of the line-of-sight 
detection method of the present invention, which has been described in 
detail, can be expected. These modifications belong to the technical 
scope of the present invention. 

All cameras used can be highly sensitive to near-infrared ray and a 
fixed-focus lens can be used by stopping down its aperture as much as 
possible to increase a focus depth. 

That is, by creating a state in which a head is always in focus even if the 
head moves significantly, the camera can have capabilities of always 
taking the head even if the direction of the camera is fixed. 
[0024] The second camera 12 may come into focus only in a narrow 
range scale because the camera is generally scaled up significantly. 
However, depending on uses, a manual-focus lens is sufficient. That is 
the case when movement of the head is limited for a short time. For 
example, when a user or subject sits on a chair, the head will not move 
significantly if the position of seat is set. Since it is sufficient, in this 
case, to be in focus in a narrow range of about 10 cm, the camera 
becomes usable when the user or subject manually focuses the lens or a 
power-zoom lens is focused once by a handy switch or a remote control. 
If the camera is moved by body such as by hand or by a handy switch or 
a remote control in a system with an actuator and once adjusted in a 
direction and at a position where eyes 1 image can be taken, the camera 
becomes capable of measuring a line-of-sight. The aforementioned 
system can be produced at a comparatively low price. Also in this case, 
the three-dimensional position of a pupil is required as information for 
line-of-sight determination. 
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[0025] In addition, if the Hne-of-sight also needs to be detected even 
when the head moves significantly and a manual-focus lens or fixed- 
focus lens cannot be focused, an automatic-focus lens or power-zoom 
lens must be used. The automatic-focus lens is generally controlled by 
5 using images of the camera. If an image has high contrast, the image 
can be used for controlling, but if an image has low contrast due to such 
as an attached band-pass filter, a power-zoom lens can be used and 
controlled from outside. 

[0026} In the aforementioned method, it was assumed that a relation 

10 between 9 and |r| is linear and passes through the origin. 

However, since an optical system and a visual axis of an eyeball are 
generally shifted slightly from each other, when a subject gazes at O, 
corneal reflection does not necessarily appear in the center of pupil 
image. That is, a visual axis does not necessarily pass through the 

15 origin. This situation must be corrected so that high-precision line-of- 
sight detection can be realized. As a corrective method, when 
calibrating a line-of-sight, it is necessary not only to make a subject 
gaze at a point somewhere, but also to measure |r| and <J) ! when the 
subject is made to gaze at O. That is, it is necessary to make the 

20 subject gaze at least at these two points to calibrate the line-of-sight. 

In this case, after shifting r obtained when gazing at another point so 
that a vector r (shift component) connecting the center of corneal 
reflection and the pupil center obtained when making a subject gaze at 
O becomes a zero vector (origin correction), |r| and <J>' are determined to 

25 perform calibration. Also in real-time line-of-sight detection, after 
making a similar correction of obtained r, |r| and $ are determined and 
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used to calculate the line-of-sight. 

[0027] A three-dimensional view-point measurement device of the 
present invention determines the three-dimensional position of a view- 
point by determining a convergent point of the line-of-sights, as well as 
performing conventional detection of the direction of the line-of-sight 
The present device is a non-contact line-of-sight detection device that 
can simultaneously detect the line-of-sights of both eyes without contact 
while allowing significant movement (for example, 30 cm) of the head 
forward and backward, or right and left from face images containing 
both eyes captured while illuminating the whole face with light sources 
provided near the aperture of two cameras apart from the face of a 
subject and capturing by these two cameras, and detects three- 
dimensional positions of two pupils by triangulation from the face 
images captured by the two cameras, further analyzes in detail 
surroundings of two pupils detected in each of two face images, and 
detects two precise pupil centers and two corneal reflection centers to 
detect the line-of-sights of two eyes. An intersection of these two line- 
of-sights is the view-point and three-dimensional view-point 
measurement can be performed. 
Embodiment 1 

[0028] As shown in Fig. 9, two cameras (camera 101, camera 102) are 
placed apart from a face 107 of a subject and light sources 105 and 106 
of the same wavelength are provided substantially inside an aperture or 
near the aperture of lenses 103 and 104 of these cameras to illuminate 
the whole face 107. The two cameras (camera 101, camera 102) are 
driven synchronously to turn on the light sources 105 and 106 mounted 
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on the two cameras alternately in sync with an odd field and an even 
field. 

As shown in Fig. 10, when the light source 105 mounted on the 
one camera 101 is turned on in the odd field, a phenomenon in which 
5 pupils are taken brighter than surroundings in images of the odd field of 
the one camera 101 occurs (bright pupil: Fig. 11 (left)), and another 
phenomenon in which pupil images are taken darker than surroundings 
in images of the odd field of the other camera 102 occurs (dark pupil: 
Fig. 11 (right)). 

10 Similarly, when the light source 106 mounted on the other 

camera 102 is turned on in the even field, dark pupils (Fig. 12 (left)) and 
bright pupils (Fig. 12 (right)) are observed in images of the even field of 
the one camera 101 and the other camera 102 respectively. 
[0029] Since other parts than the pupils cancel out each other when an 

1 5 image difference is produced by basically subtracting dark pupil images 
obtained from each camera from continuous bright pupil images, only 
pupil parts can easily be detected. 

To speed up pupil detection processing, in an initial image, 
approximate whereabouts of two pupils are detected by analyzing pixels 

20 at intervals in the whole image. Then, a window is provided around 
each of the detected approximate pupils (two squares in Fig. 1 3), and 
from the next frame onward, only images inside the windows are 
analyzed or only images inside the windows are captured from the 
cameras to shorten the time for image analysis. 

25 All pixels are basically analyzed in detail when analyzing 

images inside the two windows and the pupil centers and corneal 
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reflection centers of both eyes are detected accurately to determine the 
line-of-sights (Fig. 14). 

[0030] A CMOS sensor that is high-definition and capable of random 
access or window cutting is used as an image sensor to speed up data 
transmission from the image sensor to an image processing portion. 
This enables more frequent detection of coordinates of the pupil centers 
and corneal reflection centers. By applying a moving average of 
obtained line-of-sights or view-point data, they can be made more stable. 

Furthermore, by shortening a time interval between two 
continuous images a difference of which should be produced, 
displacement of pupil image between two images are made more 
difficult to occur, contributing to improvement in the precision and 
detection rate of not only view-points and line-of-sights, but also three- 
dimensional pupil positions. This facilitates detection of pupils based 
on differences, and at the same time, substantially enables 
synchronization of two cameras even if the head moves and pupil 
images move quickly within a camera frame. 

Incidentally, though two cameras are still used, other methods of 
improving synchronization shifts between cameras can yet be 
considered by, for example, increasing the number of light sources, 
using an optical filter, or making dedicated cameras. 
Embodiment 2 

[0031] Each of the two cameras can basically detect the line-of-sights 
of both eyes. Now, as shown in Fig. 15, for the line-of-sight of one 
eye, three-dimensional coordinate of the pupil in the world coordinate 
system constructed when camera calibration of two cameras is 
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performed is estimated from pupil coordinate obtained from the two 
cameras. Now, only the camera on the left is noticed. A camera- 
pupil vector in the world coordinate system is determined from three- 
dimensional coordinate of the pupil estimated and coordinate of the 
5 camera determined by camera calibration. If the line-of-sight vector is 
defined as a vector directed from the pupil to the view-point, the angle 
formed by the camera-pupil vector and the line-of-sight vector can be 
represented by 0 and <J> in the diagram, where 0 is, in the range of about 
±30 degrees, approximately linearly related to |r| of a vector r directed 

10 from the corneal reflection center to the pupil center in the image shown 
in Fig. 16 (0 = k|r|, where k is a constant). <j> is also related to §\ which 
indicates the direction of r, like <|> = <J>\ The view-point G is determined 
as an intersection of the line-of-sight and the visual object. 

If the plane where the visual object exists is unknown, an 

1 5 intersection of the line-of-sight vectors of both eyes is the view-point G 
[0032] In the above processing, since the line-of-sight vector of one eye 
can be obtained from each of the two cameras, an average vector of 
these vectors may be defined as the final line-of-sight vector. If 
corneal reflection disappears when 0 becomes larger and, instead, a 

20 white part of the eye starts to shine, virtually making it impossible to 
detect a line-of-sight from a relative position relation of the pupil center 
and the corneal reflection center (for example, when viewing to the right 
of the camera on the right in Fig. 15), a method must be devised such as 
using only a line-of-sight vector determined by the right camera because 

25 it is difficult to detect the line-of-sight using the left camera. 

When viewing further to the right, even the right camera cannot 
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detect corneal reflection. In this case, an approximate direction of the 
line-of-sight is detected from ellipticity of the pupil. If 9 > 30° or 
corneal reflection cannot be detected for some reason (for example, 
when spectacle reflection or eyelashes overlap corneal reflection), 
5 ellipticity shall be used. Though a method of using ellipticity is 

problematic in view of precision as mentioned above, this method is 
effective when viewing in a direction sharply deviating from the front 
and no precision is needed. 

[0033] A reason why the line-of-sight direction can be detected from 
10 relative coordinates of the corneal reflection and pupil centers is 
described below. 

Characteristics of corneal reflection> 

Corneal reflection is represented as shown in Fig. 1 9 (left) because the 
incident angle and reflection angle are the same when light emitted from 

15 a light source 121 (point light source) is reflected by a reflection plane 
to enter a camera. If, here, the object is not a plane, but a spherical 
surface, the situation can be represented as shown in Fig. 19 (right). In 
this case, when viewed from a camera 120, the light source 121 appears 
as if it were at a contact point 122 of a tangent plane 125 of a spherical 

20 surface and a spherical surface 124. The incident angle and reflection 
angle are equal relative to a straight line connecting a center 123 of the 
spherical surface and the contact point 122. 

[0034] Here, if the spherical surface (cornea of an eyeball) moves in 
space relative to the camera 120 and the light source 121, as shown in 
25 Fig. 20 (left), reflection of the light source 121 moves in the image 
when viewed from the camera 120 because the visual angle from the 
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camera changes. As described above, if the spherical surface is 
considered typically as a corneal spherical surface, it is apparent that a 
corneal reflection image moves in the image as the cornea moves due, 
for example, to movement of the head. Furthermore, if the position of 
5 the light source 121 or the position or direction of the camera 120 is 
different, it can easily be imagined that the position of corneal reflection 
image varies in the camera image. 

[0035} Now, the case is considered in which the position of the light 
source and that of the camera agree (Fig. 20 (right)). In this case, a 

10 beam directed from the light source to the spherical surface and that 
returned from the spherical surface to the camera agree. Such a 
condition is superior to the case in which the light source and the 
camera are at different positions. This is because, if the position of the 
light source and that of the camera do not agree, as shown in Fig. 21 

15 (left), a corneal reflection image moves in the image when moving in 
the depth direction relative to the light source and camera, but if they 
agree (Fig. 2 1 (right)), when the corneal sphere moves at a fixed angle 

» 

relative to a photographing direction of the camera, a corneal reflection 
image is advantageously taken at the same position in the image. 
20 Therefore, if the position of the light source and that of the camera agree, 
only rotation of surroundings of the camera (light source) causes a 
problem about the corneal reflection image. 

[0036] <A reason why the angle formed by the camera-pupil vector and 
the line-of-sight vector is known from the size |r| between the corneal 
25 reflection center and the pupil center in a photographed image and a 
method thereof> 
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It is assumed, as shown in Fig. 22, that an eyeball moves within 

* 

a field of view of a camera for line-of-sight detection. Since the three- 
dimensional position of the pupil is known with a stereo camera, it is 
assumed that the pupil moves while maintaining the distance to the 
5 camera and light source constant. At this point, it is however assumed, 
that the angle 0 formed by the camera-pupil vector and the line-of-sight 
vector does not change while moving. Here, however, in the interests 
of simplicity, it is assumed that the pupil (eyeball) moves on a 
horizontal axis of the camera. It is to be noted at this moment that the 

4 

10 physical relationship between a corneal reflection image and a pupil 
image viewed from the camera does not change at all. When the pupil 
moves on a circumference around the camera (lens, light source), a 
relative distance between the corneal reflection image and the pupil 
image in the image does not change. 

15 [0037] On the other hand, the case is considered, as shown in Fig. 23 

(left), in which the starting points of the line-of-sight vectors are the 
same (the same three-dimensional position of the pupil), but the 
direction vectors are different. In this case, as shown in the diagram, 
the position of corneal reflection changes. It is apparent that, when 

20 viewed from the camera, as the angle formed by the line-of-sight vector 
relative to the camera-pupil vector increases, that is, 0 increases, the size 
|r| between the cornea reflection center and the pupil center in the image 
increases. As Fig. 23 (right) shows, when the eye is directed to the 
camera, the corneal reflection center is essentially positioned at the 

25 pupil center and |r| = 0. 

[0038] Therefore, there is a simply increasing relation passing through 
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the origin shown in Fig. 24 between the angle 0 formed by the camera- 
pupil vector and the line-of-sight vector and the size |r| of the vector r 
from the corneal reflection center to the pupil center. This relation 
depends on the subject due to a difference of the optical system of the 
eyeball. Determining the 0-|r| relation can be said to be calibration of 
the line-of-sight of the present technique. Furthermore, since tiiere is 
normally a shift between the optical axis and visual axis of the optical 
system of the eye, the above relation needs to be corrected because the 
origin is not necessarily passed through. These points will be 
described later. 

■ 

[0039] In both of Fig. 23 (right) and Fig. 23 (left), it is to be noted that 9 
and |r| do not change when only the direction (photographing direction 
around th£ point O) of the camera changes. Therefore, information on 
the direction of camera is not needed and simply a pupil and corneal 
reflection need to be taken in a camera frame. Moreover, as described 
using Fig. 22, a fact that |r| still remains constant is reemphasized even 
when the eyeball moves around the point O, as long as the distance from 
the point O to the pupil center P is maintained constant while 
maintaining 9 constant. 

[0040] However, if the distance (OP) between the point O of the camera 
position and the pupil center P changes, |r| changes even though 9 does 
not change. This is because, if the camera lens is a fixed-focus lens, as 
the distance between the camera and an object increases, the object 
taken as an image becomes smaller. Thus, this situation must be 
corrected. The correction is considered by using the fact that, as 
described above, the same 9-|r| relation is always realized regardless of 
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the angle formed by the camera-pupil vector and the optical axis of the 
camera. Now it is assumed that, to facilitate illustration, the pupil 
exists at a position where, though the distance from the camera to the 
pupil center changes, the direction of the line-of-sight does not change, 
as shown in Fig. 25. As is evident from the diagram, when the 
distance viewed from the camera changes, the size |r| between the pupil 
center and corneal reflection center taken by the camera changes (In this 
case, coordinate of the pupil center in the image naturally change. 
However, this needs not to be considered and it is sufficient to focus on 
a relative position relation between corneal reflection and the pupil 
center.). 

[0041] Now, it is however assumed that, because |r| changes due to a 
change of the distance OP, as described above, a camera exists that 
corrects this situation and always takes an object with the same size. If, 
as shown in Fig. 26, the object (here an eyeball) is drawn by changing 
the size thereof depending on die distance OP, |r| can be maintained 
constant independent of the distance OP For a fixed-lens camera that 
cannot take an object with the same size when the distance changes, |r| 
in the image may be corrected depending on the distance. However, a 
relation between the distance OP and |r| for correction must be examined 
in advance. In contrast to calibration of the line-of-sight, however, this 
is a kind of camera correction and has nothing to do with a subject. 
Here, OP is calculated from the P coordinate determined by the stereo 
camera. 

In addition, in Fig. 25, an optical path connecting the light 
source and the pupil center is actually not a straight line, but is refracted 
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at a boundary surface between air and the front of cornea and at a 
boundary surface between the back of cornea and aqueous humor. As 
is evident from Fig. 25, as the distance OP changes, the optical path 
connecting the light source and the pupil center changes and the 
changed path passes through a different corneal position. Since the 
incident angle of light to the boundary surface changes depending on 

■ 

the corneal position, the refraction angle at the boundary surface also 
changes. Therefore, even if the aforementioned camera correction is 
made, |rj is still subtly affected by the distance OP. |r| is corrected by 
using, for example, refractive index and cornea curvature radius 
information of the standard human cornea and aqueous humor, as 
required. 

[0042] <Angle (<|)) of the line-of-sight vector around the camera-pupil 
vector and calibration of the line-of-sight> 

Incidentally, § must be clarified, in addition to 0, to determine a 
line-of-sight vector. However, as described above, (j) and the angle (J)' 
formed by the vector r directed from the corneal reflection to the pupil 
center in the image and the horizontal axis (axis of abscissas in Fig. 16) 
of the camera must basically be equal. 

Thus, it is suggested that no calibration is needed for the angle 
(<|)) of the line-of-sight vector around the camera-pupil vector. If this 
assumption is correct and the 9-|r| relation shown in Fig. 24 is linear and 
passes through the origin, it becomes possible to perform line-of-sight 
calibration by only making a subject gaze at a point somewhere. 
Embodiment 3 

[0043] With capabilities of simultaneously measuring the line-of-sights 
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of both eyes, it becomes possible to measure an angle of convergence 

i 

(angle formed by the line-of-sights of both eyes) and a view-point in the 
depth direction (intersection of the line-of-sights of both eyes), 
indicating the distance to an object being gazed at. As a result, the 
5 view-point in three-dimensional space can be known. 

This suggests an application to evolved 3D display and the like. 

■ 

If the eyes are not focused on a normal 3D display, it is not possible to 
view clear images. Thus, a watcher is forced to watch in an unnatural 
way against the normal human visual function that the eye focus does 

10 not change when the angle of convergence changes. However, by 
using an evolved 3D display that, even if the focus is shifted in the 
depth direction, comes into focus accordingly, more natural three- 
dimensional visual appreciation will be experienced. 

In this type of display, by providing spatial positions of the 

1 5 eyes and focal positions of the eyes, the display can be adjusted so that 
images can selectively be displayed accordingly in a direction of two 
eyes, and at the same time, the focal positions of the eyes actually come 
into focus. Information (three-dimensional positions of two pupils and 
point of intersection of the line-of-sights of both eyes, 3D view-point) 

20 required for such adjustments can be obtained from the present system. 

Three-dimensional pointing using the line-of-sights also becomes 
possible. 

That is, in a normal 2D display and view-point mouse by one 
eye, the view-point mouse is simply used for selecting windows 
25 arranged on a two-dimensional plane, but in the above-described full- 
scale 3D display, windows are arranged in 3D space and a window 
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hidden behind can immediately be displayed on top of other windows 
by a three-dimensional view-point. In this way, the present system can 
also be used for three-dimensional pointing. 
Embodiment 4 

[0044] With the configuration described above, three-dimensional 
positions of two pupils can be measured. The pupil is the source of a 
line-of-sight and shows from where a user is gazing (not where a user is 
gazing at). 

As an application, it is supposed that there is a screen and a wall 
in front of a user in a virtual reality or actual world, and it is not possible 
to view beyond the screen (Fig. 17). However, it is supposed that a 
camera to take images displayed on the screen is placed beyond the 
screen and there is the wall where there is no screen. 

At this point, the screen can be caused to operate as if the screen 
were a window and outside of the window were viewed. Namely, if 
the head moves (Fig. 18), part of images taken by a wide-angle camera 
is caused to be displayed on the screen accordingly. In this way, 
required images can be displayed in a form full of such realism. 

In the concept of "Pointing system by pupil movement" 
proposed by the present inventors, movement of a pupil within the 

♦ 

camera frame is simply caused to be reflected in cursor movement, but 
according to the present system, since the three-dimensional position of 
a pupil is known, movement of the head in the depth direction relative 
to the screen can be caused to be reflected. 

[0045] Arrangement of the screen and camera depends on application. 
In any case, in an image that is displayed by enlarging it, parts that are 
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hidden behind and invisible can be seen by moving the head as if 
peeping out of the window. Thus, an image to be displayed on the 
screen does not have to be always an object nearby, and an object in the 
distance or a micro-object taken by enlarging it may be acceptable. 
5 Parts to be seen can be changed by moving the head right and left, or up 
and down (rotation is also permissible), and further by moving the head 
forward and backward, the zoom ratio is significantly changed so that 
an object can be seen from a favorite angle with a desired zoom ratio. 

Therefore, when used in combination with a PC, if a three- 
10 dimensional object (for example, a molecular model) is displayed on the 
PC screen and to be seen by rotating it, a mouse for a desktop computer 
is normally used, but by using the present device, the object can be 
freely moved in a natural atmosphere only by rotating or translating the 
head. 

15 According to the present system, if unchanged, the same image 

will be seen by both eyes. If this is inconvenient, images of different 
cameras may be caused to enter the right and left eyes by using a 3D 
display and two cameras. 
Embodiment 5 

20 [0046] The present application is useful in an environment where 
something should be hidden from other people such as a security system. 
For example, to make the display invisible to other people even if they 
look into when using an ATM system or the like in a bank, the present 
method can be used to control the display so that the display is visible 

25 only from the two pupil positions of the user or a nearby direction. 

Similarly, to do PC work that should not be glanced at when 

r 
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other people are present around (for example, in Shinkansen), by 
combining a display that can make content visible only from a specific 
direction with the present method, the content can be made visible only 
to a person who operates the PC. 

For the above purposes, a display unit such as a liquid crystal 
display that has a narrow angle of visibility and can be controlled is 
suitable. 

The present method has another advantage. Because the 
display can be made to have a strong directivity, strong light can be sent 
into the eye while reducing power consumption. 
Embodiment 6 

[0047] Further, a field of electronic books can be considered as another 
embodiment. An advantage of the electronic books is that, by using a 
dedicated device, a page can be turned only by one button click. 
However, a manual operation is still needed. By combining such a 
dedicated device for electronic books with view-point detection means, 
for example, when the view-point moves significantly in a vertical 
writing text from left to right, the page can be caused to be 
automatically turned. Or, by providing a button area for page turning 
at a lower left corner of the screen or device, the page can be caused to 
be automatically turned when the view-point is aligned with the button 
area. Using such a device, it becomes possible to read a book without 
using a hand while lying in a bed or the like. This makes reading more 
comfortable and further increases advantages of electronic books. This 
will make very effective means for four limbs paralytics. The view- 
point detection means of the present invention can be made compact 
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and such an embodiment can be carried out easily. 

* 

Here, the description has assumed a dedicated device for 
electronic books to be combined with the view-point detection means, 
but the means is still valid for notebook computer usage, and by 
embedding two cameras around a liquid crystal display, the line-of-sight 
can be used for selecting an active window or moving a high-speed 
mouse using the line-of-sight of the user. 
Industrial Applicability 

[0048] A line-of-sight detection method and device of the present 

« 

invention can be widely used in the field of monitoring human behavior 
or as line-of-sight input means in a human life environment or work 
environment. Application will be planned in the field of 
manufacturing electronic devices and in an environment of using 
electronic computers. 

[0049] The three-dimensional view-point measurement device of the 
present invention has the applicability shown below: 

1. With capabilities of simultaneously measuring the line-of-sights of 
both eyes using at least two cameras and two light sources, it becomes 
possible to measure an angle of convergence (angle formed by the line- 
of-sights of both eyes) and a view-point in the depth direction 
(intersection of the line-of-sights of both eyes), indicating the distance 
to an object being gazed at. As a result, the view-point in three- 
dimensional space can be known (point of intersection of the line-of- 
sights of both eyes, 3D view-point). 

2. Three-dimensional positions of two pupils can be measured, A 
pupil is the source of a line-of-sight and shows from where a user is 
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gazing (not where a user is gazing at). Therefore, the device is 
effective in a system in which from where being gazed at must exactly 
be known. 

» 

(Conventionally, this has had to be realized by attaching a 
magnetic sensor or the like to the head and measuring movement of the 
head, but whereabouts of pupils could not be exactly measured, in 
addition to attachment of a sensor to the head) 

3. The present system is easy to realize because it has no actuator and 
two cameras are fixed. 

If the camera has high sensitivity, the aperture of the camera can 
be made smaller, and the light source can easily be made compact by 
producing high-intensity light sources. 

4. Because the line-of-sight can be detected by both of two cameras, the 
line-of-sight precision can be further improved by processing such as 
averaging detection results obtained from these cameras. 
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